Upgrade vendored packaging lib #12300

sbidoul · 2023-09-29T20:13:45Z

Upgrade our vendored copy of packaging, remove all references to LegacyVersion and LegacySpecifier.

Closes #12063
Closes #11715
Closes #11445
Closes #7650
Closes #9613
Closes #10098

TODO

handle resolution with candidates with invalid versions
handle resolution with candidates with invalid requirements
handle environments containing installed distributions with legacy versions
handle environments containing installed distributions with invalid specifiers in their dependencies

sbidoul · 2023-09-29T21:20:23Z

@pypa/pip-committers this is a quick try at upgrading packaging.

The tests that need fixing may reveal a few interesting things.

In particular I wonder if we should worry about 920c4fc. When the pip-test-package-0.1.tar.gz sdist is in the find-links path, pip now fails with Invalid version test-package-0.1.

sbidoul · 2023-09-29T21:43:11Z

And the last failing test (test_install_special_extra[Hop_hOp-hoP-hop-hop-hop]) eludes me for now. It succeeds on 3.11 and 3.12 but fails on 3.10 and below.

tests/unit/metadata/test_metadata_pkg_resources.py

tests/functional/test_install_extras.py

src/pip/_vendor/vendor.txt

pradyunsg · 2023-10-06T19:04:41Z

I just tried a couple of quick changes to the pkg_resources metadata backend to make it respect PEP 685 normalisation. A quick check reflects that this does behave correctly -- I'll wait for the CI check on this though, as well as a round of reviews before moving forward with it.

pradyunsg · 2023-10-06T19:32:48Z

Quoting @pfmoore from the discussion above...

The problem is that setuptools.DistInfoDistribution makes the extras attribute a list of extras normalised by safe_extra. But pip has canonicalised the requested extra, and it can't re-constitute the non-canonical value that's in extras (nor should it). I don't think this is easily fixable without a fairly extensive rewrite of our extra handling (or a fix to pkg_resources to store normalised extra names, which, to be fair, isn't required by PEP 685).

Ugh, this is a nuance that I missed digging into this head-first. Indeed, dependency_map[safe_extra(ext)] is how pkg_resources handles names. :/

pfmoore · 2023-10-06T20:44:58Z

Nice! Looks like you may have fixed it @pradyunsg 🙂 I hadn't thought to go the whole way and manage getting the extras from the metadata directly (effectively copying the importlib.metadata approach for pkg_resources). That's a much more localised change than I'd feared we might need.

pradyunsg · 2023-10-07T00:02:39Z

Ohkie, this last commit is an attempt at simplification -- we might want to leave it out if it isn't all-green, since it's unlikely I'd get a chance to look at this again before Monday morning.

pradyunsg · 2023-10-07T19:32:36Z

I've taken the liberty to mark this as ready for review.

pradyunsg · 2023-10-07T19:33:26Z

As I do that though... I'm realising that one thing that we ought to validate is whether there are suboptiomal behaviours/failure modes, when given a legacy version or specifier; coming via the index page, a built wheel, a project's metadata and various other data sources that pip interacts with.

sbidoul · 2023-10-08T11:28:38Z

This is not ready indeed. There is work to do to ignore invalid versions in indexes (e.g. pip install pytz fails because there are old invalid versions on PyPI), and ignore distributions with invalid version specifiers in their dependencies. And handle invalid installed versions. And probably more...

notatallshaw · 2023-10-08T12:21:07Z

Other fails include pip install nltk, pip install apache-airflow[all], and pip install pandas (I raised similiar issues with rip recently which I beleive also vendors the latest version of Packaging: prefix-dev/rip#27, prefix-dev/rip#38, and prefix-dev/rip#25)

FYI in PR #12327 I am currently filtering out invalid file name and version name packages as Pip collects them in that particular scenario, not sure if that's the going to be the prefered way to do it.

sbidoul · 2023-10-08T15:50:12Z

Yeah, I now remember why I said I would not have time to do this before the release. I somehow had managed to forget about #12300 (comment) when I created this PR.

This seems to be way too far reaching to do so late in the release cycle. So I think I'll postpone it.

pfmoore · 2023-10-08T15:56:37Z

There is work to do to ignore invalid versions in indexes (e.g. pip install pytz fails because there are old invalid versions on PyPI), and ignore distributions with invalid version specifiers in their dependencies. And handle invalid installed versions. And probably more...

Are there issues tracking these (that we could put into the 24.0 milestone)? If we don't have some means of tracking what's needed to get this landed, I fear we're going to be in the same situation next release - in spite of no-one liking it, vendoring upgrades tend to get ignored until late in the release cycle...

We do this by adding a version_str property to BaseDistribution, which allows us to access the version without parsing it.

We do this to avoid needlessly building a sdist when we have determined that a wheel has invalid metadata.

Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

Dependencies should be more precise than requirements;

The pkg_ressources metadata backend does not suffer from invalid metadata in this test case.

tacaswell · 2024-05-10T17:47:50Z

This breaks pip on the CPython maintenance branches as the version reported by Python ends with a + and nothing else

ERROR: Exception:
Traceback (most recent call last):
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_internal/commands/install.py", line 377, in run
    requirement_set = resolver.resolve(
                      ^^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
    result = self._result = resolver.resolve(
                            ^^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 427, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 239, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/resolvelib/resolvers.py", line 229, in _get_updated_criteria
    for requirement in self._p.get_dependencies(candidate=candidate):
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 247, in get_dependencies
    return [r for r in candidate.iter_dependencies(with_requires) if r is not None]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 243, in iter_dependencies
    for r in requires:
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_dists.py", line 222, in iter_dependencies
    elif not extras and req.marker.evaluate({"extra": ""}):
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/packaging/markers.py", line 252, in evaluate
    return _evaluate_markers(self._markers, current_environment)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/packaging/markers.py", line 158, in _evaluate_markers
    groups[-1].append(_eval_op(lhs_value, op, rhs_value))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/packaging/markers.py", line 116, in _eval_op
    return spec.contains(lhs, prereleases=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/packaging/specifiers.py", line 568, in contains
    normalized_item = _coerce_version(item)
                      ^^^^^^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/packaging/specifiers.py", line 36, in _coerce_version
    version = Version(version)
              ^^^^^^^^^^^^^^^^
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/packaging/version.py", line 200, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
pip._vendor.packaging.version.InvalidVersion: Invalid version: '3.12.3+'

Bisected to 47a8480 specifically.

psf-chronographer bot added the bot:chronographer:provided label Sep 29, 2023

sbidoul force-pushed the packaging-upgrade branch from d9546f5 to 63897d3 Compare September 29, 2023 21:15

sbidoul mentioned this pull request Sep 29, 2023

Deprecate legacy versions and version specifiers #12063

Closed

sbidoul added this to the 23.3 milestone Sep 29, 2023

sbidoul commented Sep 29, 2023

View reviewed changes

tests/unit/metadata/test_metadata_pkg_resources.py Outdated Show resolved Hide resolved

mayeut mentioned this pull request Sep 30, 2023

New release pypa/packaging#718

Closed

sbidoul commented Sep 30, 2023

View reviewed changes

tests/functional/test_install_extras.py Show resolved Hide resolved

sbidoul force-pushed the packaging-upgrade branch 2 times, most recently from af9fbb4 to fd89669 Compare October 1, 2023 12:31

sbidoul mentioned this pull request Oct 1, 2023

Release 23.3 #12296

Closed

sbidoul force-pushed the packaging-upgrade branch from 828a7d3 to 3af04b6 Compare October 1, 2023 13:14

mayeut reviewed Oct 1, 2023

View reviewed changes

src/pip/_vendor/vendor.txt Outdated Show resolved Hide resolved

sbidoul force-pushed the packaging-upgrade branch 3 times, most recently from f731c58 to 62e9e9b Compare October 6, 2023 13:42

pradyunsg force-pushed the packaging-upgrade branch from 0b6ebfa to 7db6cb3 Compare October 7, 2023 00:01

pradyunsg marked this pull request as ready for review October 7, 2023 19:29

sbidoul removed this from the 23.3 milestone Oct 8, 2023

sbidoul and others added 23 commits May 4, 2024 02:06

Allow uninstallation of dist with legacy version

73f6744

We do this by adding a version_str property to BaseDistribution, which allows us to access the version without parsing it.

Refactor test

0563132

Add failing test for pip list in presence of legacy version

27807fb

Fix pip list in presence of installed legacy versions

0529c04

Add failing test for pip freeze in presence of legacy version

93e52cf

Fix pip freeze in presence of legacy version

a28d13a

Add failing test for pip show of legacy version

be652aa

Fix pip show of legacy version

8f97eb5

Upgrade packaging to 24.0

08ae751

Add news

196d536

Add test sdist that requires invalid version

83e77bf

Ignore all candidates of a version when one has invalid metadata

b458d3e

We do this to avoid needlessly building a sdist when we have determined that a wheel has invalid metadata.

Simplify

350c980

Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>

Don't create a list when an iterator is sufficient

f4b821c

Rename version_str to raw_version for consistency with raw_name

c50290e

Add failing test for pip show of dist with legacy specifier

00edcf4

Add iterator of raw dependencie (Requires-Dist) to BaseDistribution

f24fa66

Fix pip show of dist with legacy dependency specifiers

d85efce

Clarify warning messages

4052f6a

Dependencies should be more precise than requirements;

Clarify test packages README

edc98cd

Add a failing test for upgrading a distribution with an invalid version

4bb5085

Add a failing test for upgrading a distribution with invalid metadata

c51fb4f

Fix test_show_require_invalid_version with pkg_resources backend

84ad55a

The pkg_ressources metadata backend does not suffer from invalid metadata in this test case.

pradyunsg force-pushed the packaging-upgrade branch from dc2b212 to 84ad55a Compare May 4, 2024 01:07

pradyunsg merged commit 4caa6c3 into pypa:main May 4, 2024
28 checks passed

ichard26 mentioned this pull request May 5, 2024

Add warning about the MyProject @ git+file format not working #12125

Closed

sbidoul deleted the packaging-upgrade branch May 6, 2024 11:28

tacaswell mentioned this pull request May 10, 2024

pip does not work with development versions of CPython due to parse error on version string #12691

Closed

1 task

github-actions bot locked as resolved and limited conversation to collaborators May 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade vendored packaging lib #12300

Upgrade vendored packaging lib #12300

sbidoul commented Sep 29, 2023 •

edited

Loading

sbidoul commented Sep 29, 2023 •

edited

Loading

sbidoul commented Sep 29, 2023

pradyunsg commented Oct 6, 2023

pradyunsg commented Oct 6, 2023

pfmoore commented Oct 6, 2023

pradyunsg commented Oct 7, 2023

pradyunsg commented Oct 7, 2023

pradyunsg commented Oct 7, 2023

sbidoul commented Oct 8, 2023 •

edited

Loading

notatallshaw commented Oct 8, 2023 •

edited

Loading

sbidoul commented Oct 8, 2023

pfmoore commented Oct 8, 2023

tacaswell commented May 10, 2024

Upgrade vendored packaging lib #12300

Upgrade vendored packaging lib #12300

Conversation

sbidoul commented Sep 29, 2023 • edited Loading

sbidoul commented Sep 29, 2023 • edited Loading

sbidoul commented Sep 29, 2023

pradyunsg commented Oct 6, 2023

pradyunsg commented Oct 6, 2023

pfmoore commented Oct 6, 2023

pradyunsg commented Oct 7, 2023

pradyunsg commented Oct 7, 2023

pradyunsg commented Oct 7, 2023

sbidoul commented Oct 8, 2023 • edited Loading

notatallshaw commented Oct 8, 2023 • edited Loading

sbidoul commented Oct 8, 2023

pfmoore commented Oct 8, 2023

tacaswell commented May 10, 2024

sbidoul commented Sep 29, 2023 •

edited

Loading

sbidoul commented Sep 29, 2023 •

edited

Loading

sbidoul commented Oct 8, 2023 •

edited

Loading

notatallshaw commented Oct 8, 2023 •

edited

Loading